K 



DOCUMENT RESUME 



ED 390 462 



JC 960 041 



AUTHOR 

TITLE 

PUB DATE 
NOTE - 



PUB TYPE 



Schueler, Paul 

Sleeping with the Enemy: Making Outcomes As s es sment 
Work in the Classroom. 

Feb 94 

8p . ; In: Proceedings of the National Conference on 
Successful Coll ege Teaching (18th , Orlando , Florida, 
February 26-28, 1994); see JC 960 033. 

Reports - Descriptive (141) — Speeches/Conference 
Papers (150) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOI Plus Postage. 

Chemistry; ^College Outcomes Assessment; ’'College 
Planning; Community Colleges; ’'Course Objectives; 
Course Organ izat ion ; ’'Curriculum Development ; 
’'Educational Assessment; ’^Educational Objectives; 
Needs Assessment; Program Evaluation; Science 
Instruction; Two Year Colleges 
Gardiner (Lion F) ; New Jersey 



ABSTRACT 

In 1991, the New Jersey Board of Higher Education's 
College Outcomes Evaluation Program (COEP) was canceled due to lack 
of state funding. Despite the cancellation, 17 community colleges 
formed the New Jersey County College Project on General Education to 
develop goals, objectives, and assessment criteria for general 
education. The Project grounds its efforts in the ideas of Lion F. 
Gardiner. Gardiner argues that in assessment planning, the first 
component of a successful course program should be the outcomes, or 
the student knowledge, skills, and achievements that are desired. 
Then, processes (i.e., courses, classroom techniques, and lab 
experiments) can be devised to achieve the goals, and inputs (i.e., 
human anH nonhuman resources) can be acquired. Using this process for- 
planning chemistry courses, the following two outcome objectives were 
developed and revised: (1) the student will exhibit an understanding 

of the scientific method: observe, make inference, classify and 
organize information, analyze and synthesize data, draw conclusions, 
and communicate those conclusions in writing; and (2) the student 
will be able to utilize information from the enormous and rapidly 
expanding chemical literature, both in libraries and in on-line 
interactive databases. The key to a positive and successful outcomes 
assessment program is a clear set of objectives and ownership and 
control by faculty and by local institutions. Contains 11 references. 
(TGI) 
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SLEEPING WITH THE ENEMY: 

MAKING OUTCOMES ASSESSMENT WORK IN THE CLASSROOM 

Dr. Paul Schueler, 

Raritan Valley Community College 



TTie Outcomes Assessment movement is having a major impact on colleges across the naUon. In New Jersey, the 
sute-run Outcomes Assessment program was canceled a few years ago when its ftmdmg was eliminauw. 
Remarkably, community college faculty and administrators arc continuing the effort voluntarily, both at local 
and on a statewide basis. What is motivating us to expend time and effort on a program that has been 
Drived as burdensome, judgmental, and "unacademic*? We are convinced that a clear, academically sound 
statement of measurable student goals and objecUves can lead to dramatic improvements m course and program 
content and methodology. The author wUl describe the ongoing effects of this process on the AS Chemistry 
program, and on freshman and sophomore level (General and Orgamc) chemistry courses. 

Outcomes Assessment, mandated by Sute Boards of Higher Education or their equivalents, by Governors, or by 
Sute Ugislatures, is having a major impact on colleges across the nation. In New Jersey, the state-run Outcomes 
Assessment program was canceled a few years ago when its funding was eliminated. Remarkably, commumty 
college faculty «nd administrators are continuing the effort voluntarily > both at local campuses and on a statewi e 
basis. What is motivating us to expend time and effort on a program that at times has been perceived as 
burdensome, judgmental, and unacademic? We are convinced that a clear, academically sound statement of 
measurable student goals and objectives can lead to dramatic improvements in course and program content and 
methodology. I will describe the ongoing effects of this process on freshman and sophomore level (General and 
Organic) chemistry courses. 

Traditionally, content (already overwhelming and ever-growing), classroom methodology (predominantly lecture, 
with experiments designed to verify previously introduced theory), and assessment techniques (commonly objective 
>.Y«iiiin«tinn» ) have dominated chemistry teaching at the introductory level, as they have for most content-based 
courses. Whether we want or intend it, such factors have impIiciUy driven student outcomes and colored not only 
what we expect from our students, but also what they expect from the course. Tbs result has contributed to the 
mutual sense of alienation in science education (Tobias, 1990) and has been critierzed not only by experts in 
education and learning styles, but by the scientific establishment as well (American Chemical Society, 1989; 
American AssocisUion for the Advancement of Science, 1990; National Science Foundation, 1990). 

Carefully thought-out goals and objectives, if developed and owned by the teacher, often necessitate profound 
ch«ngi.<! m the content, teaching methods, and assessment techniques of the course. The content of introductory 
chemistry courses is a hotly debated topic, in the literature as well as at conferences and department meetings 
(Bodner, 1992; Spencer, 1992). It is a subject beyond the scope of this paper, but there is a consensus that a 
reduction in the quantity of material presented is an essential prerequisite to further reform. As fw as teachmg and 
■f yMwnwjit methods are concerned, the modem pedagogical toolbox is filled with methodologies which cm be 
employed to foster and measure the desired outcomes; cooperative and collaborative techniques, critical t hinkin g 
exercises, demonstrations and simulations, discovery or guided inquiry laboratories, information retrieval and 
analysis, and writing (papers, lab reports, and essay questions). 

In this paper, I will draw on wy experiences with Outcomes Assessment in New Jersey, as a participant at the 
State-wide level, as a member of a college task force, as a developer of program goals and objectives in the Science 
and Engineering Department, and finally as the teacher of two courses in particular: General Chemistry and Organic 
Chemistfy. The focus will be on changes instituted by my colleagues and myself largely in response to goals and 
objectives that we had developed. 



Outcomes Assessment in New Jersey , 

The New Jersey Board of Higher Education created the College Outcomes Evaluation Program (COEP) in June, 
1985. Due to the economic do mtam in the sutc, the program was canceled for lack of sute funding m 1991. 
Although COEP was administered by the New Jersey Department of Higher Education and funded from the state 
budget, as determined by the Governor and the Legislature, from the outset, the program was designed and 
controlled largely by faculty and administrators from the colleges in the sute. The higher education community 
in New Jersey is quite complex: large public (Rutgers, New Jersey Institute of Technology, University of Medicine 
and Dentistry) and private (Princeton, Fairleigh Dickinson) universities; baccalaureate and Mastere level colleges, 
both public (the state college system, centrally administered by the Department of Higher Education) and private; 
and the county colleges, independent two-year associate degree granting institutions, controlled and funded by the 
counties, but deriving a significant amount of funding from the state. All sectors were included in COEP, both as 
mandated (and in some cases reluctant) participants and as contributors to its development and final form. 

COEP had five principle components: fl) a General Intellectual Skills (GIS) test; (2) general education; (3) major 
subject programs; (4) student development; and (5) community impact. Of these, the GIS test was the most 
innovative and controversial. The COEP committee decided to develop its own instrument, intended to be 
administered to students at the beginning of the freshman year and again at the end of the sophomore year, in order 
to assess ihe effect of their college experiences on their intellectual performance (the value-added approach). The 
test was developed by the Educational Testing Service (ETS); at the insistence of C and with the assistance 
of many college and university faculty members, ETS produced an exam that is quite unlike the familiar multiple 
choice exam for which it is famous. The student is given one of a number of tasks. Typically, a task will consist 
of material for the student to read and answer questions about. The major assignment is to write a paper or report. 
The tasks cover a wide range of topics, from history to sociology to art to poetry to science. The exam was 
intended to serve as a means of evaluating institutions rather than individual students. Although many were 
suspicious of the use to which it might be put, there was general agreement among teachers that it represented an 
ixmovative approach to assessment. ETS now owns the test and is marketing it to college and umversity systems. 
Dr. Angela Bodino, of the English department, and I have received a PEW grant in order to investigate the use of 
the GIS model ay a way of evaluating students* critical thinking abilities and as a way of presenting material to 
enhance critical thinking. 

While COEP was in effect, Raritan Valley Community College (RVCC) was actively involved in all of its 
components, with particular emphasis on the general education and major program evaluations. I was a member 
of the college-wide committee which reviewed program goals and objectives, and I helped develop such goals and 
objectives for the AS in Chemistry and the AS in General Science programs. Like most faculty members, I 
formulated and revised student outcome objectives for the courses that 1 taught. After the demise of the state-wide 
COEP program, RVCC has continued its own outcomes assessment at the general education, program and course 
level. This voluntary continuation is motivated partly by the perceived benefits of outcomes evaluation to the 
institution, to the faculty, and to the students, and partly by the uncertain but realistic prospect that the sute would 
reimpose a similar mandate when and if funds became available. 

Perhaps the most remarkable recent development has been the formation of the New Jersey County College Project 
on General Education by seventeen of the nineteen county colleges for the express purpose of developing goals, 
objectives, and assessment criteria for general education on a state-wide basis. I am a member of one of its four 
ta^ forces, staffed by volunteer faculty and administrators from each of the colleges. Goals and objectives for 
general education have been written; we are currently engaged in developing assessment criteria and, if necessary, 
instruments for these objectives. The results will be distributed to the member institutions, undoubtedly debated 
and revised, until a final product is written. Each college will be free to adopt all or part of the general education 
assessment package. The democratic, participatory, grass-roots character of this effort is a continuation and 
expansion of the best spirit of COEP and, if successful, will be a model for further sUte-wide assessment effort. 

Developing Goals. Obieettves. and Assessment Instruments 

There is an extensive body of literature dealing with outcomes assessment; this paper is not intended as a review 
or overview of the field. In my experience, one of the most lucid and practically oriented experts in the field is 
Professor Lion F. Gardiner of Rutgers University, who was a major consultant to COEP and is now assisting the 
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County College project. Two of his most useful pubUcations «e a H«idbook prov.dmg pt^UcaJ assistance to 
SSrs tiying to develop assessment pl«is (Gardiner. 1991) «id a resource paper hstmg (with bnef^mmen^) 
a selection of publications covering a wide range of educational theoiy and practice m this area (Gardmer, 1992). 
Both are available from the New Jersey Department of Higher Education. 



Some of the flavor of Gardiner’s approach can be sensed in an anonymous quote he uses: "If you don t 
know where you’re going, you’ll probably end up someplace else." 



According to Gardiner, there are three crucial organuational components of a successful course or prog^: mputs 
(human and nonhuman resources), processes (courses, classroom techniques, lab experiments), ttd outcomes 
(student knowledge, skills, and acyevements). In an operational sense, these three co^nents are listed m their 
temporal order one starts with the inputs, applies the processes, and concludes with the outcomes. In 
however the order is reversed; the first component to be determined should be the student outcome (what result 
do we want to obtain). Processes which can produce those results can then be devised, and the mputs necessary 
to support those processes are obtained. 

This order seems at first glance to be a simple, even self-evident approach, but our collective uaching ex^rience 
abounds with examples that arc anything but consistent with it. In my cwn field of chemistry, for exa^le, most 
of us haven’t (until recently) spent much time thinking or talking about what we expect of our undergi^uate 

students, particularly in the freshman and sophomore years. Our implicit long-term goal seems to ve turn 

them aU into chemists like us. this despite the fact that, even at research universities, over mnety 
students in introductory chemistry courses are not chemistry majors, and even those who are more likely to be 
heading for medical or dental schools than chemistry graduate programs. The processes of the cour^ (^tent. 
method, laboratory experiments, evaluation methods) are geared to this vague and implicit outcome, wbeh for most 
of the students is both unobtainable and inappropriate. 



There are. of course, student outcomes for an introductory chemistry course wbeh are based on the acquisition of 
knowledge, or on the development of skills wbch are well-formulated and appropriately evaluated m tradiUo^ 
course formats. However, there are some outcomes wbch, once they are chosen and committed to r^uire that 
we carefully .-r«mine our courses and make some profound revisions m the teaebng processes. In the last part ot 
the paper. 1 will summarize a few of the revisions that I and my colleagues have made m response to two objecuves 
developed for the AS Oiemistry program. 

Objective 1. The student wUl exhibU an understanding of the scientific method: observe, make inferences, classify 
and organize information, analyze and synthesize data, draw condusions, and communicate those condustons m 

writing. 

At first glance, tbs seems to be a straightforward, if ambitious, objective. In the iwub course, however, the 
methods employed hardly promote, much less guarantee, the desired outcome. The Scientific Method (m capitals) 
is presented m a lecture, perhaps with illustrative examples, for the students to memorize. In the laboratoiy, 
students perform experiments presented m a cookbook format, with numerical results inserted mto blanks on 
preprinted daU sheets wbch arc tom out and handed m. Although the student may become skilled at certam lab 
techbques (wbch fbfills an entirely different objective), he or she may not understand the methods of science m 
any meanmgfiil way, nor, m the event that he or she docs understand, has there been any opportumty to demonstrate 
or exbbit that understandmg. 

Perhaps the most promismg innovation m the purmiit of understandmg m the laboratory is the Discovery or Guid^ 
Inquiry approach to the teaebng of mtroductory chemistry laboratories (Pavelich & Abrahams, 1979; Allen, et. al., 
1986, Ricci & Ditzler, 1991). As developed by Ditzler and bs colleagues at Holy Cross, the teaebng of general 
chemistry is driven by the laboratories: topics are usually mtroduced first m the lab where the students are given 
a problem or concept to explore, develop (with guidance from the mstructor) hypotheses, experiment with different 
methods, decide how to analyze and report their data, and synthesize the entire class results into a conclusion that 
leads to jn«ght mto the underlymg theoretical prmciples. The challenge to chemists is to develop opm-ended 




experiments which arc within the capabilities of beginning chemistry students, feasible with the equipment and 
supplies available, and yet interesting and profound enough to stimulate productive thought. 

One of the simplest and most intriguing of the experiments developed at Holy Cross is "Penmes," an expenmenl 
which 1 have used at the beginning of the chemistry sequence in conjunction with a unit on measurement. The lab 
revolves around the question: "What happens to a penny as it ages?" Before going into the lab, the class as a group 
develops hypotheses (e.g., it loses mass), supported by reasons (e.g., frictional wear), and develops an expenmental 
protocol to test the hypotheses (e.g., weighing pennies sorted by their mint year). Wheii data has been gathered, 
the class discusses methods of anaiymg the data (plotting mass against mint year). In this case, the penmes show 
no clear tread in gain or loss from year to year, but there is an abrupt drop in mass in 1932. This generates 
a new set of hypotheses to explain this discontinuity, which in turn must be tested (e.g., the densities can be 
determined to chock for a change in composition, which is indeed the case; most of the copper was removed from 
the penny in 1982). If time permits, more extensive investigations can be performed. The beauty of this 
experiment is in its deceptive simplicity and accessibility even for the beginning student. Its openendedness and 
potential for fairly sophisticated elaboration are hallmarks of a well •conceived discovery experiment. 

A second example is one that I have devised for a different group, second semester students in Organic Chemistry. 
One of the early experiments in a unit called Organic (Qualitative Analysis (in which the students identify unknown 
compounds) is one in which the solubility characteristics of classes of compounds is explored. Tr^itionally, the 
students arc presented with the classification scheme in a lecture with some theory and reaction equations to support 
it. They are then sent into the lab to verify the solubility patterns for known compounds and to classify an unknown 
compound by determining its solubility. In restructuring this experiment as a discovery lab, I have follow^ the 
lead of my colleague Roger Johnson, a plant biologist, who devised a similar restnicturing for an experiment in the 
classification of twigs and branches based on a number of physical criteria. 

In my variation, the class is given a set of known compounds (typically about ten distinct functional group classes) 
and a set of solvents (five or six aqueous and nonaqueous solvent mixtures). Tht class as a group develops 
hypotheses, trying to predict, based on their chemical knowledge and experience, the solubility behavior of the 
various compounds. They discuss ai? experimental approach, collect data in the lab, and analyze the results. With 
some guidance from the teacher, if all goes well, the class produces a classification scheme essentially identical to 
the one in the textbooks, but as a product of its own investigations. 

Objective 2 : The student will be able to utilize information from the enormous and rapidly expanding chemical 
literature, both in libraries arui in ondine interactive databases. 

This objective is realized through the completion of three categories of activities. 

1) Literature search paper: 

Students in the second semester of General Chemistry were given the following assignment: 

Write a paper describing the work of a prominent chemist of the last two centuries. Students will work 
in groups of two or three, choosing a chemist whose contributions are regarded as important. A reasonable 
starting point might be the list of Nobel Prize wiimers in chemistry, or the chemists whose work is 
described at various places in your textbook. Once 1 have approved your group and choice of a subject, 
you will find the necessary sources, put together a rough draft (which I will be glad to look at, if you 
wish), and then write the paper for submission. Each group will submit one paper; each member of the 
group will get the same grade for the assignment. Describe the major themes of the chemist*s scientific 
work. If possible, discover how he or she regarded his or her place in the profession. Was there anything 
unusual in the way the subject became a chemist, approached the field, or conducted his or her career? 
Describe the impact of the chemist's work on the science of chemistry and the chemists who followed. 
You must cite at least three sources, only one of which can be an encyclopedia article (no textbooks 
allowed). If possible, at least one source must be written by the subject of the paper. When the 
assignment was first announced, many of the students reacted with sonoe dismay (a paper in a science 
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class'’) Most of them expressed positive feelings about the assignment and their papers by the end of the 
semester, ranging from grudging acceptance to great enthusiasm. One experience particularly stands out: 
two female students chose to write about Dorothy Crowfoot Hodgkin, a pioneer in the development of 
X-ray crystallography who won the Nobel Prize for Chemistry in 1964. Their investigation of her career 
gave them a vivid lesson in both the opportunities and the obstacles facing women in science. 

2) Individual experiments: , 

As the final laboratoiy experiment in Organic Chemistiy D. the students were asked to select a synthesis 
from the literature, to adapt it if necessaiy to the equipment, supplies, and scale of the available resources, 
to perform the experiment, and to write up their results. Problems chosen ranged from the simple to the 
complex; a few students were ambitious tnd creative enough to put together a multistep synthesis. In 
addition to gaining familiarity with the literature and the search procedure, most students were gratified 
by a sense of ownership not felt in doing a textbook experiment. 

3) Comparative articles: . . 

When buckminsterfullerenes (’’buckybaUs") were first discovered and charactenzed. they cau^t the 
imagination not only of chemists and other scientists, but of the general population as well. My colleague 
and feUow chemist SheUa CanceUa was inspired to use these appealing molecules to stimulate discussion 
of the various OKxies of scientific reporting in the literature. She handed out three articles: one from a 
refereed scientific journal (Science), one from a trade magazine (Chemical and Engineering News), and 
one from the popular press (TIME). aU describing the discovery and stnictural features of buckyballs. The 
class discussed the similarities and differences in the treatments of the basic facts. Many scientific 
developments of the last few years lend themselves to this treatment, e.g.. cold fusion. AIDS research, the 
Hubble space telescope, global waiming. 

In conclusion, despite the doubts and fears that outcomes assessment can engender in the t each in g faculty, a 
carefully crafted set of objectives can trigger important improvements in course content, process, and evaluation. 
The lesson learned from the New Jersey experience has been that ownership and control by faculty and by local 
institutions is a key element in a positive and successful program. 
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